Gesture activated home appliance

ABSTRACT

Apparatus for operating home appliances using gesture recognition are disclosed. An image receiver receives a continuous stream of images of a gesture made within the image receiver&#39;s field of view. An image processor is connected to the image receiver for sampling the received stream of gesture images to form a gesture image sequence, and for recognizing whether the gesture image sequence corresponds to one of a predefined set of gestures. If the input gesture is recognized as being one of the predefined gestures, an operations processor, which is connected to the image processor, identifies a home appliance operation associated with the recognized gesture. An appliance controller is connected to the operations processor for causing the predefined home appliance operation to be performed. Home appliances and methods for operating home appliances using gesture recognition are also disclosed.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.11/206,432 filed on Aug. 18, 2005 now U.S. Pat. No. 7,236,611, entitled“Gesture Activated Home Appliance” which is a continuation of U.S.application Ser. No. 09/965,906 filed on Sep. 28, 2001, entitled“Gesture Activated Home Appliance,” issued as U.S. Pat. No. 6,937,742 onAug. 30, 2005, which are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to gesture recognition systems. Moreparticularly, the present invention relates to systems and methods foroperating a home appliance using gesture recognition.

BACKGROUND OF THE INVENTION

Voice activated technology is a rapidly developing area of the computerworld. Today, many devices incorporate voice activation technology socertain functions of the device can be performed based on voicecommands. For example, many home appliances are equipped withvoice-activated technology as to allow a consumer to orally command, forexample, a lighting system to power on by using voice commands. Such afeature is particularly advantageous when a person cannot manuallyactivate a device because their hands are occupied or the device switchis in an inconvenient place.

For example, one may be carrying groceries into a house and is unable tomanually activate the light switch, consequently, if the lighting systemin the house has voice activated technology therein, the person maysimply say, for example, “lights on” to activate the lights. Thistechnology saves the person the trouble from having to put the groceriesdown, manually activate the lights, and then pick up the groceries. Theability to use voice-activated technology, of course, presumes that theuser is able to speak. Consequently, those persons who cannot speakcannot take advantage of voice-activated technology.

Gesture recognition is another rapidly developing area in the computerworld. Gesture recognition is not based on voice commands but, rather,allows a device to recognize certain gestures. The inventors, however,are not aware of any gesture recognition systems that are used inconjunction with the operation of home appliances. Therefore, there is aneed for systems and methods that use gesture recognition to operatehome appliances.

BRIEF SUMMARY OF THE INVENTION

The present invention satisfies the aforementioned need by providingsystems and methods for using gesture recognition to operate a homeappliance. In one embodiment of the present invention, an apparatusrecognizes a gesture and then causes a home appliance operationassociated with the gesture to be performed. For example, a firstgesture might be associated with powering on a home appliance and asecond gesture might be associated with powering off the home appliance.In this manner, a user can simply gesticulate to operate a homeappliance rather than operating the appliance manually.

Apparatus according to the invention for operating a home appliance caninclude an image receiver, an image processor, an operations processor,and an appliance controller. The image receiver, which can be a videocamera, for example, receives a continuous stream of images of a gestureperformed within the image receiver's field of view and provides a videosignal to the image processor. The video signal contains informationcorresponding to the continuous stream of gesture images. The imageprocessor receives the video signal and recognizes the gesture based onthe information contained in the video signal.

The image processor can include an image sampler and an imagecomparator. The image sampler can be used for sampling the continuousstream of gesture images to form a discrete sequence of gesture images.The image comparator can then compare the discrete sequence of gestureimages to each of one or more stored discrete gesture sequences, andrecognize the gesture based on the comparison.

The operations processor is coupled to the image processor foridentifying a home appliance operation associated with the gesture. Theoperations processor can identify the home appliance operation bycomparing the recognized gesture with each of a predefined set ofgestures. Each of the predefined gestures is associated with arespective home appliance operation.

The appliance controller is coupled to the operations processor forcausing the home appliance operation to be performed. The apparatus caninclude a switch that is coupled to the appliance controller and isadapted to be coupled to the home appliance. The switch is responsive tothe appliance controller for causing the home appliance operation to beperformed. The appliance controller can be a voltage generator forgenerating a voltage signal, or a pulse generator for generating avoltage pulse, that causes the home appliance operation to be performed.

The apparatus can also include a performance indicator for providing anindication that the home appliance operation has been performed. A datastore, readable by the image process, or storing data representative ofthe gesture and of the home appliance operation.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

Other features of the present invention are further apparent from thefollowing detailed description of the embodiments of the presentinvention taken in conjunction with the accompanying drawings, of which:

FIG. 1 is a block diagram of a device used to operate a home appliancein accordance with the present invention;

FIGS. 2A and 2B illustrate the device of FIG. 1 connected to homeappliances; and

FIG. 3 is flowchart illustrating a method of operating a home appliancein accordance with the present invention.

DETAILED DESCRIPTION OF THE PRESENT INVENTION

The present invention is directed to systems and methods for operating ahome appliance using the principles of gesture recognition technology.In this manner, a person may turn on an oven, for example, by performinga recognizable gesture associated with turning on the oven. Similarly,the person may also turn off the oven, set the oven temperature, turn onthe oven light, or cause the oven to perform any other operation byperforming a gesture associated with respective operation. Consequently,a user can gesticulate to operate a home appliance rather than manuallyactivate a switch or a button to operate the home appliance.

The present invention relates to apparatuses and methods for operating ahome appliance. The apparatus may be a computer, in whole or in part,specially constructed to perform the method of the invention, or it maybe a general purpose computer selectively activated or reconfigured by acomputer program stored in the computer. General purpose computingmachines may be used with programs written in accordance with theteachings herein, or a specialized computing apparatus may beconstructed to perform the method described herein.

FIG. 1 is a block diagram of a device 100 for operating a home appliance(not shown) using gesture recognition in accordance with the presentinvention. As shown, the device 100 includes an image receiver 110 forreceiving a continuous stream of gesture images. The image receiver 110may be any PC-based camera or the like without departing from theprinciples of the present invention. The image receiver 110 transmitsthe video signals to an image processor 120 by way of a cable 112. Thevideo signals may be sent in any format such as, for example, MPEG3format or the like without departing from the principles of the presentinvention.

The image processor 120 forms a discrete sequence of gesture imagesbased on the continuous stream of gesture images it receives from theimage recover 110. In this manner and in one embodiment of the presentinvention, the image processor contains an image sampler 120A forforming a discrete sequence of gesture images from the received gestureimages by sampling the continuous stream of images. In other words, theimage sampler 120A segments the continuous stream of images into aseries of frames or “snapshots.”

The image processor 120 also derives positional data of the gestures ineach frame, and compiles the positional data into a respective framedata set. The positional data in the frame data set is compared, in theimage comparator 120B, to each of a set of stored sequences ofpositional data making up one or more recognizable gestures i.e.,gestures already known to the system. Any gesture, such as, for example,a gesture associated with American Sign Language can be stored in a datastore 150. In this regard, the comparator 120B can search the data store150 to match the received gesture with a stored gesture by using thepositional data of the data frame set of the received gesture to thepositional data of one of the known sequences stored in the data store150. The image comparator 120B, based on the comparison, determineswhether the user within the field of view 115 of the image receiver 110has performed a recognizable gesture.

Thus, the image receiver 110 can receive a continuous image of agesture, which can be, for example, a person moving an arm horizontallyacross the chest. The image sampler 120A segments the gesture intoframes. The image processor 120 then obtains positional data from eachframe of the gesture sequence and compiles the data in a data frame set.Then, the image comparator 120B compares the positional data of the dataframe set to the positional data of the recognizable gestures stored inthe database.

Once a gesture is recognized (i.e., the system determines that thereceived image corresponds to a recognizable gesture), an operationsprocessor 130 connected to the image processor 120 identifies apredefined home appliance operation associated with the recognizedgesture. In one embodiment of the present invention, the operationsprocessor 130 determines, based on the recognized gesture, which of aplurality of predefined home appliance operations is to be performed. Inthis manner, the operations processor can access the data store 150 todetermine the predefined home appliance operation associated with therecognized gesture.

As shown and for example, one recognizable gesture 116 might includemotioning the letter “z” with the index finger of the hand. In thisregard, once the device 100 recognizes this gesture, the operationsprocessor 130 searches the data store 150 to determine which homeappliance operation is associated with motioning the letter “z” with theindex finger. For example, this gesture might be associated with turningon the appliance. Another recognizable gesture, such as moving one armhorizontally across the chest, for example, might be associated withturning off the home appliance.

An appliance controller 140 is connected to the operations processor 130for causing the predefined home appliance operation to be performed. Inone embodiment of the present invention, the appliance controller isconnected to a switch 180. In this manner, for example, if thepredefined home appliance operation is turning on the appliance, theappliance controller 140 can be configured to transmit an electricalsignal that actuates switch 180 that turns on the appliance.Consequently, the appliance controller 140 is configured to actuate thepower switch 180 that turns on the appliance if the gesture associatedwith powering on the appliance is performed and recognized by the device100. In this regard, the appliance controller 140 may be a voltagegenerator for generating a voltage signal that actuates the power switch180. Alternatively, the appliance controller 140 can be a pulsegenerator for generating a voltage pulse to actuate a switch 180 thatmay or may not be located within the appliance.

The device 100 in accordance with the present invention can also includea performance indicator 160, such as, for example, a sound generatorconnected to the appliance controller 140. The performance indicator 160indicates the predefined home appliance operation has been performed. Inone embodiment of the present invention, the performance indicator 160is a sound generator that emits, in response to the appliance controller140 causing the predefined home appliance operation to be performed, asound to notify the user that the home appliance operation has occurred.This method of indication is particular useful if the home applianceoperation cannot be visually verified, e.g. turning on an oven or thelike.

In one embodiment of the invention, each home appliance operation isassociated with a different audible sound. For example, immediatelyafter turning on the appliance, the sound generator may emit a singlebeep. Conversely, immediately after turning off the appliance, the soundgenerator may emit a double beep. This notification method provides theuser with an audible indication of what particular home applianceoperation has been performed.

FIGS. 2A and 2B each illustrates the apparatus of FIG. 1 connected to ahome appliance. As shown in FIG. 2A, lighting system 220 is connected toan apparatus 100 in accordance with the present invention. As describedabove, the apparatus 100 receives images by way of image receiver 110and transmits the images to an image processor 120 through cable 112.The image processor 120 recognizes a sequence of gesture images based onthe stream of gesture images received by the image receiver 110.

Once a gesture is recognized, an operations processor 130 connected tothe image processor 120 identifies a predefined home appliance operationassociated with that recognized gesture. An appliance controller 140 isconnected to the operations processor 130 for causing the predefinedhome appliance operation to be performed. In this manner, for example,if the home appliance operation is turning on the lighting system 220,the appliance controller 140 is configured to turn on the lightingsystem 320 using switch 229. Consequently, the appliance controller 140transmits a signal by way of cable 230 to the lighting system 220 if thegesture associated with powering on the lighting system 220 is performedand recognized. A performance indicator 160 connected to the appliancecontroller 140 emits, in response to the appliance controller 140turning on the lighting system 220, an indication to notify the userthat the lighting system 220 is on.

Similarly, as shown in FIG. 2B, an oven 260 is connected to an apparatus100 in accordance with the present invention by cable 265. Again, asdescribed above, the apparatus 100 receives images of a gesture 116 byway of image receiver 110 and transmits the images to image processor120 through cable 112. Image processor 120 recognizes a sequence ofgesture images based on the stream of gesture images received by theimage receiver 110.

Once a gesture is recognized, an operations processor 130 connected tothe image processor 120 identifies which predefined home applianceoperation is associated with the recognized gesture. An appliancecontroller 140 is connected to the operations processor 130 for causingthe predefined home appliance operation associated with the recognizedgesture to be performed. In this manner, for example, if the homeappliance operation is turning off the oven 260, the appliancecontroller 140 is configured to turn off the oven 260 by way of switch264. Consequently, the appliance controller 140 causes the oven 260 tobe turned on if the gesture associated with powering on the oven 260 isperformed and recognized. A performance indicator 160 connected to theappliance controller 140 indicates, after the appliance controller 140is turned on, that the oven 260 is on.

The operations processor 130 can also include programming and logic thatenable it to provide more sophisticated signals to the home appliance.For example, where the home appliance is an oven, the operationsprocessor can be programmed to send any of a number of various signalsto the oven, thereby causing the oven to perform any number ofoperations. For example, a first recognizable gesture can be associatedwith turning the oven on, a second gesture can be associated withsetting the oven temperature, a third gesture can be associated withsetting a cooking timer, a fourth gesture can be associated with turningon the oven's internal light, and a fifth gesture can be associated withturning the oven off. Each such recognizable gesture would be associatedwith the corresponding operation in the system's data store.

Depending on the gesture the recognized, the operations processor cansend a signal to the oven's computer that emulates the signal the oven'scomputer would have received is the user had pushed the appropriatebuttons on the oven itself. For example, pushing the oven on buttonmight cause a certain signal to be sent to the oven's computer. Theoven's computer then turn the oven on. Accordingly, the operationsprocessor of the invention can send the same signal directly to theoven's computer, thereby causing the oven's computer to turn the ovenon.

FIG. 3 is flowchart illustrating a method of operating a home appliancein accordance with the present invention. In the method, a user performsa gesture within the field of view of an image receiver, such as acamera. Thus, the image receiver receives a continuous stream of gestureimages at step 300. The system uses the continuous image stream todetermine whether the gesture that the user is making is one of a set ofone or more recognizable gestures. In one embodiment of the presentinvention, the gesture is recognized by first forming a discretesequence of gesture images from the continuous image stream at step 310.Then, using positional data in a data frame set derived from each frame,the discrete sequence of gesture images is compared, at step 320, to aset of one or more stored sequences of discrete gesture images. Based onthe comparison, the system determines, at step 330, whether theperformed gesture is one of the set of recognizable gestures.

At step 340, the recognized gesture is identified with a predefined homeappliance operation. The recognized gesture is compared to a set ofstored gestures, each gesture being associated with a predefined homeappliance operation. Once the predefined operation identified with therecognized gesture is found, the system causes the predefined homeappliance operation to be performed at step 350. At step 360, anindication can be provided to indicate to the user that the operationhas been performed.

Thus, there have been described systems and methods for operating a homeappliance using gesture recognition technology. Though the invention hasbeen described with reference to a device that is external andconnectable to a home appliance, it is it contemplated that such adevice may be integrated into such an appliance. Consequently, thedevice in accordance with the principles of the present invention shouldnot be limited to a device that is separate from the home appliance.Those skilled in the art will appreciate that numerous other changes andmodifications can be made to the preferred embodiments of the invention,and that such changes and modifications can be made without departingfrom the spirit of the invention. It is intended, therefore, that theappended claims cover all such equivalent variations as fall within thetrue spirit and scope of the invention.

1. An apparatus for controlling an appliance, the apparatus comprising:means for receiving an image stream, the image stream comprising agesture; means for recognizing the gesture from the image stream; meansfor identifying a home appliance operation associated with the imagestream, wherein the gesture is associated with a distinct home applianceoperation; and means for causing the distinct home appliance operationto be performed, wherein means for causing the distinct home applianceoperation to be performed comprises: means for accessing a data store todetermine the distinct home appliance operation associated with thegesture, and means for causing the distinct home appliance operation tobe performed via a switch coupled to an appliance controller wherein themeans for causing the distinct home appliance operation to be performedis configured to actuate the switch to turn on a home appliance if therecognized gesture is associated with the distinct home applianceoperation associated with the home appliance.
 2. The apparatus of claim1, further comprising: means for segmenting the image stream in to aseries of frames; means for deriving positional data corresponding tothe gesture from the frames; and means for comparing the positional datacorresponding to the gesture to each of a set of stored sequences ofpositional data to determine if the gesture is a recognized gesture. 3.The apparatus of claim 1, further comprising: means for sampling theimage stream to form a gesture image sequence; means for comparing thegesture image sequence to at least one stored gesture sequences; andmeans for recognizing the gesture based on the comparison.
 4. Theapparatus of claim 1, wherein the identify means comprises means forcomparing the recognized gesture with each of a predefined set ofgestures.
 5. The apparatus of claim 1, further comprising means fornotifying a user that the distinct home appliance operation has beenperformed.
 6. The apparatus of claim 5, wherein the notifying meansfurther comprising means for generating a sound for notifying a userthat the distinct home appliance operation has been performed.
 7. Theapparatus of claim 6, wherein the generating means further comprisesmeans for generating a distinct sound for each of a plurality of homeappliance operations to be performed.
 8. The apparatus of claim 1,wherein the causing means further comprises means for generating avoltage signal for causing the distinct home appliance operation to beperformed.
 9. The apparatus of claim 1, wherein the causing meansfurther comprises means for generating a voltage pulse for causing thedistinct home appliance operation to be performed.
 10. A method forcontrolling an appliance, the apparatus comprising: receiving a gesture,wherein the gesture is recognized from an image stream and is associatedwith a distinct home appliance operation; causing the distinct homeappliance operation to be performed in response to receiving thegesture, wherein causing the distinct home appliance operation to beperformed in response to receiving the gesture comprises: accessing adata store to determine the distinct home appliance operation associatedwith the gesture, and causing the distinct home appliance operation tobe performed via a switch coupled to an appliance controller.
 11. Themethod of claim 10, further comprising: segmenting the image stream into a series of frames; deriving positional data corresponding to thegesture from the frames; and comparing the positional data correspondingto the gesture to each of a set of stored sequences of positional datato determine if the gesture is a recognized gesture.
 12. The method ofclaim 10, further comprising: sampling the image stream to form agesture image sequence; comparing the gesture image sequence to at leastone stored gesture sequences; and recognizing the gesture based on thecomparison.
 13. The method of claim 10, further comprising notifying auser that the distinct home appliance operation has been performed. 14.A system for controlling an appliance, the system comprising: a memorystorage; and a processing unit coupled to the memory storage, whereinthe processing unit is operative to: receiving a gesture, wherein thegesture is recognized from an image stream and is associated with anappliance operation; causing the appliance operation to be performed inresponse to receiving the gesture wherein causing the applianceoperation to be performed in response to receiving the gesturecomprises: accessing a data store to determine the appliance operationassociated with the gesture, and causing the appliance operation to beperformed via a switch coupled to an appliance controller.
 15. Thesystem of claim 14, wherein the processing unit is further operative to:segment the image stream in to a series of frames; derive positionaldata corresponding to the gesture from the frames; and compare thepositional data corresponding to the gesture to each of a set of storedsequences of positional data to determine if the gesture is a recognizedgesture.
 16. The system of claim 14, wherein the processing unit isfurther operative to: sample the image stream to form a gesture imagesequence; compare the gesture image sequence to at least one storedgesture sequences; and recognize the gesture based on the comparison.17. The system of claim 14, wherein the processing unit is furtheroperative to cause a notification to be sent to a user that theappliance operation has been performed.
 18. The system of claim 14,wherein the processing unit being operative to causing the applianceoperation to be performed in response to receiving the gesture comprisesthe processing unit being operative to generate at least one of thefollowing: a voltage signal and a voltage pulse for causing theappliance operation to be performed.